Methods of Parallel Experimental Design of Online Automatic Tuning and their Application to Parallel Sparse Matrix Data Structure

Automatic tuning is one of key technologies in high performance computing, where parallel processing is essential. In this paper, we propose some methods of parallel experimental design for online automatic tuning of parallel programs. In parallel processing, two kinds of tuning should be investigated. One is local tuning, which optimizes local tuning parameters on each processor, and the other is global tuning, that affects executions of all processors. This paper deals with local tuning, and proposes three methods. The first method we consider is SEO (Serial Experiments Once), where each processor once measures the performance of all the candidates. The second method is PEO (Parallel Experiments Once), where the candidates are evaluated in a distributed manner over the parallel processors. Each processor observes the performance of a subset of the candidates that are assigned to it. Then the performance results are collected, and the candidate of the best performance is chosen. The third method is MPEO (Modified PEO), where each processor can replace the candidate chosen by PEO if it performs badly. Pros and cons of those methods are discussed, and results of application of those methods to an online optimization of parallel sparse matrix data structure are shown.